Ranking Hyperlinks Approach for Focused Web Crawler

The World Wide Web is growing rapidly and many search engines do not cover all the visible pages. Therefore, a more effective crawling method is required to collect more accurate data. In this paper, we introduce an effective focused web crawler containing smart methods. In text analysis, similarity measurement applies to different parts of the Web pages including title, body, anchor text and URL tokens. It can increase the relevance and quality of the Web pages pointed to by target URLs. To enhance the accuracy of crawling, decay concept is used to determine the optimal order in which the targeted URLs are visited. In this measurement, two kinds of threshold are used to limit the crawler to the effective web pages. Finally, to provide sorting URLs, priority equation is used. Our method shows significant performance improvements in crawling efficiency over previous focused crawling. Keywords—decay concept, focused web crawler, priority equation, similarity space model.